{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7388fcde",
   "metadata": {},
   "source": [
    "# Simulation Experiment Empirical Validation\n",
    "\n",
    "**The Problem**: You've built a TinyTroupe simulation, but how do you know if it accurately reflects real-world behavior?\n",
    "\n",
    "**The Solution**: This validation system compares your simulation results against empirical data to give you a confidence score.\n",
    "\n",
    "*Note: data here is fictitious and for demonstration purposes only.*\n",
    "\n",
    "## Real-World Example: E-commerce Checkout Optimization\n",
    "\n",
    "**Scenario**: Your company is considering a new premium checkout flow. You have real customer data from the current system and want to validate your TinyTroupe simulation predictions before making a $2M investment.\n",
    "\n",
    "**The Stakes**: If your simulation is wrong, you could lose customers and revenue. If it's right, you could increase conversions by 40%.\n",
    "\n",
    "Let's see how close your simulation gets to reality..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "477d7c72",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-14T20:20:05.012549Z",
     "iopub.status.busy": "2025-07-14T20:20:05.011552Z",
     "iopub.status.idle": "2025-07-14T20:20:06.837936Z",
     "shell.execute_reply": "2025-07-14T20:20:06.835530Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "!!!!\n",
      "DISCLAIMER: TinyTroupe relies on Artificial Intelligence (AI) models to generate content. \n",
      "The AI models are not perfect and may produce inappropriate or inacurate results. \n",
      "For any serious or consequential use, please review the generated content before using it.\n",
      "!!!!\n",
      "\n",
      "Looking for default config on: c:\\Users\\pdasilva\\repos\\TinyTroupe\\examples\\..\\tinytroupe\\utils\\..\\config.ini\n",
      "Found custom config on: c:\\Users\\pdasilva\\repos\\TinyTroupe\\examples\\config.ini\n",
      "TinyTroupe version: 0.5.1\n",
      "Current date and time (local): 2025-07-15 23:29:09\n",
      "Current date and time (UTC):   2025-07-16 02:29:09\n",
      "\n",
      "=================================\n",
      "Current TinyTroupe configuration \n",
      "=================================\n",
      "[OpenAI]\n",
      "api_type = openai\n",
      "azure_api_version = 2024-08-01-preview\n",
      "model = gpt-4o-mini\n",
      "reasoning_model = o3-mini\n",
      "embedding_model = text-embedding-3-small\n",
      "max_tokens = 16000\n",
      "temperature = 1.7\n",
      "freq_penalty = 0.1\n",
      "presence_penalty = 0.1\n",
      "timeout = 480\n",
      "max_attempts = 5\n",
      "waiting_time = 0\n",
      "exponential_backoff_factor = 5\n",
      "reasoning_effort = high\n",
      "cache_api_calls = False\n",
      "cache_file_name = openai_api_cache.pickle\n",
      "max_content_display_length = 1024\n",
      "azure_embedding_model_api_version = 2023-05-15\n",
      "\n",
      "[Simulation]\n",
      "parallel_agent_generation = True\n",
      "parallel_agent_actions = True\n",
      "rai_harmful_content_prevention = True\n",
      "rai_copyright_infringement_prevention = True\n",
      "\n",
      "[Cognition]\n",
      "enable_memory_consolidation = True\n",
      "min_episode_length = 15\n",
      "max_episode_length = 50\n",
      "episodic_memory_fixed_prefix_length = 10\n",
      "episodic_memory_lookback_length = 20\n",
      "\n",
      "[ActionGenerator]\n",
      "max_attempts = 2\n",
      "enable_quality_checks = False\n",
      "enable_regeneration = True\n",
      "enable_direct_correction = False\n",
      "enable_quality_check_for_persona_adherence = True\n",
      "enable_quality_check_for_selfconsistency = False\n",
      "enable_quality_check_for_fluency = False\n",
      "enable_quality_check_for_suitability = False\n",
      "enable_quality_check_for_similarity = False\n",
      "continue_on_failure = True\n",
      "quality_threshold = 5\n",
      "\n",
      "[Logging]\n",
      "loglevel = ERROR\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import json\n",
    "sys.path.insert(0, '..')\n",
    "\n",
    "from tinytroupe.validation import SimulationExperimentEmpiricalValidator, SimulationExperimentDataset, validate_simulation_experiment_empirically\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeaa987b",
   "metadata": {},
   "source": [
    "## The Data: Real Customer Behavior vs. TinyTroupe Simulation\n",
    "\n",
    "**Real Data**: Customers tested the current checkout flow over 3 months  \n",
    "**Simulation**: TinyTroupe agents tested the proposed premium checkout flow  \n",
    "**Question**: Can we trust the simulation to predict real customer behavior?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bb1b69da",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-14T03:49:59.634632Z",
     "iopub.status.busy": "2025-07-14T03:49:59.633630Z",
     "iopub.status.idle": "2025-07-14T03:49:59.660147Z",
     "shell.execute_reply": "2025-07-14T03:49:59.658069Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📊 Real customer data loaded (3 months of actual e-commerce behavior)\n",
      "🤖 TinyTroupe simulation data loaded (premium checkout predictions)\n",
      "💰 Predicted revenue increase: 68% per visitor\n",
      "⚠️  Question: Can we trust this simulation before investing $2M?\n"
     ]
    }
   ],
   "source": [
    "# REAL CUSTOMER DATA (3 months of actual e-commerce data)\n",
    "real_customer_data = {\n",
    "    \"name\": \"Real E-commerce Customer Data\",\n",
    "    \"description\": \"Actual customer behavior from current checkout system\",\n",
    "    \"key_results\": {\n",
    "        # Extended data with more sample points to avoid statistical issues\n",
    "        \"conversion_rate\": [0.23, 0.19, 0.25, 0.21, 0.24, 0.18, 0.22, 0.20, 0.26, 0.19, \n",
    "                           0.24, 0.22, 0.20, 0.25, 0.23, 0.21, 0.19, 0.24, 0.22, 0.20],  # 20 weekly averages\n",
    "        \"cart_abandonment_rate\": [0.68, 0.72, 0.65, 0.70, 0.67, 0.74, 0.69, 0.71, 0.64, 0.73,\n",
    "                                 0.69, 0.71, 0.68, 0.66, 0.72, 0.70, 0.67, 0.69, 0.71, 0.68],\n",
    "        \"average_order_value\": [87.50, 92.30, 85.20, 89.10, 91.80, 83.40, 88.90, 86.70, 94.20, 84.60,\n",
    "                               89.30, 91.50, 87.80, 90.20, 88.60, 86.40, 92.10, 89.70, 87.30, 90.80],\n",
    "        \"customer_satisfaction\": [3.2, 3.4, 3.1, 3.3, 3.5, 3.0, 3.2, 3.4, 3.6, 3.1,\n",
    "                                 3.3, 3.2, 3.4, 3.1, 3.3, 3.5, 3.0, 3.2, 3.4, 3.3],\n",
    "        \"overall_revenue_per_visitor\": 20.13  # Key business metric\n",
    "    },\n",
    "    \"result_types\": {\n",
    "        \"conversion_rate\": \"per_agent\",\n",
    "        \"cart_abandonment_rate\": \"per_agent\", \n",
    "        \"average_order_value\": \"per_agent\",\n",
    "        \"customer_satisfaction\": \"per_agent\",\n",
    "        \"overall_revenue_per_visitor\": \"aggregate\"\n",
    "    },\n",
    "    \"agent_justifications\": [\n",
    "        \"Too many steps in checkout - gave up halfway through\",\n",
    "        \"Payment options were confusing, wasn't sure which to choose\",\n",
    "        \"Completed purchase but the process felt unnecessarily complicated\",\n",
    "        \"Loading times were too slow, lost patience\",\n",
    "        \"Smooth experience overall, would buy again\"\n",
    "    ],\n",
    "    \"justification_summary\": \"Current checkout has friction points: too many steps, confusing payment options, slow loading times. Customers abandon due to complexity.\"\n",
    "}\n",
    "\n",
    "# TINYTROUPE SIMULATION DATA (proposed premium checkout with AI assistance)\n",
    "tinytroupe_simulation_data = {\n",
    "    \"name\": \"TinyTroupe Premium Checkout Simulation\",\n",
    "    \"description\": \"Simulation of new premium checkout with AI assistant and streamlined flow\",\n",
    "    \"key_results\": {\n",
    "        # Extended data with more sample points to match real data\n",
    "        \"conversion_rate\": [0.34, 0.31, 0.36, 0.33, 0.35, 0.29, 0.32, 0.30, 0.37, 0.31,\n",
    "                           0.33, 0.35, 0.32, 0.34, 0.36, 0.30, 0.33, 0.35, 0.31, 0.34],  # Predicted higher conversion\n",
    "        \"cart_abandonment_rate\": [0.45, 0.48, 0.42, 0.46, 0.44, 0.50, 0.43, 0.47, 0.41, 0.49,\n",
    "                                 0.44, 0.46, 0.43, 0.45, 0.47, 0.42, 0.48, 0.44, 0.46, 0.45],  # Predicted lower abandonment\n",
    "        \"average_order_value\": [102.30, 108.50, 99.80, 105.20, 110.40, 97.60, 104.10, 101.90, 112.30, 98.70,\n",
    "                               105.80, 103.40, 107.20, 101.60, 109.30, 104.70, 106.50, 102.80, 108.90, 105.10],  # Higher AOV\n",
    "        \"customer_satisfaction\": [4.1, 4.3, 3.9, 4.2, 4.4, 3.8, 4.0, 4.2, 4.5, 3.9,\n",
    "                                 4.2, 4.1, 4.3, 4.0, 4.4, 3.9, 4.1, 4.3, 4.2, 4.0],  # Higher satisfaction\n",
    "        \"overall_revenue_per_visitor\": 33.85  # 68% increase predicted!\n",
    "    },\n",
    "    \"result_types\": {\n",
    "        \"conversion_rate\": \"per_agent\",\n",
    "        \"cart_abandonment_rate\": \"per_agent\",\n",
    "        \"average_order_value\": \"per_agent\", \n",
    "        \"customer_satisfaction\": \"per_agent\",\n",
    "        \"overall_revenue_per_visitor\": \"aggregate\"\n",
    "    },\n",
    "    \"agent_justifications\": [\n",
    "        \"AI assistant helped me find exactly what I needed quickly\",\n",
    "        \"One-click checkout made the process effortless\",\n",
    "        \"Smart recommendations increased my order value naturally\",\n",
    "        \"Felt confident about my purchase with the AI guidance\",\n",
    "        \"Fastest checkout experience I've ever had\"\n",
    "    ],\n",
    "    \"justification_summary\": \"Premium checkout eliminates friction with AI assistance, one-click payment, and smart recommendations. Customers feel guided and confident.\"\n",
    "}\n",
    "\n",
    "print(\"📊 Real customer data loaded (3 months of actual e-commerce behavior)\")\n",
    "print(\"🤖 TinyTroupe simulation data loaded (premium checkout predictions)\")\n",
    "print(\"💰 Predicted revenue increase: 68% per visitor\")\n",
    "print(\"⚠️  Question: Can we trust this simulation before investing $2M?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a337453",
   "metadata": {},
   "source": [
    "## The Validation: Can We Trust the Simulation?\n",
    "\n",
    "The system compares your simulation against real data using:\n",
    "- **Statistical tests** - Are the differences significant?\n",
    "- **Semantic analysis** - Do the reasons make sense?\n",
    "- **Confidence score** - How much can we trust this simulation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "824d384a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-14T03:49:59.666379Z",
     "iopub.status.busy": "2025-07-14T03:49:59.666379Z",
     "iopub.status.idle": "2025-07-14T03:50:42.069708Z",
     "shell.execute_reply": "2025-07-14T03:50:42.064708Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🎯 VALIDATION RESULTS\n",
      "==================================================\n",
      "📊 Confidence Score: 33.2%\n",
      "📈 Simulation Quality: Statistical validation: 4/5 tests significant, average effect size: 4.739; Semantic validation: Average proximity score of 0.282; Summary proximity: 0.400; Overall validation score: 0.332\n",
      "❌ LOW CONFIDENCE - Simulation may not be reliable\n"
     ]
    }
   ],
   "source": [
    "# Run the validation - this is where the magic happens!\n",
    "import warnings\n",
    "import numpy as np\n",
    "\n",
    "# Suppress statistical warnings for cleaner output\n",
    "warnings.filterwarnings('ignore', category=RuntimeWarning)\n",
    "\n",
    "try:\n",
    "    validation_result = validate_simulation_experiment_empirically(\n",
    "        control_data=real_customer_data,\n",
    "        treatment_data=tinytroupe_simulation_data,\n",
    "        validation_types=[\"statistical\", \"semantic\"],\n",
    "        significance_level=0.05,\n",
    "        output_format=\"values\"\n",
    "    )\n",
    "    \n",
    "    print(\"🎯 VALIDATION RESULTS\")\n",
    "    print(\"=\" * 50)\n",
    "    \n",
    "    if validation_result.overall_score is not None:\n",
    "        print(f\"📊 Confidence Score: {validation_result.overall_score:.1%}\")\n",
    "        print(f\"📈 Simulation Quality: {validation_result.summary}\")\n",
    "        \n",
    "        # Quick interpretation\n",
    "        if validation_result.overall_score > 0.8:\n",
    "            print(\"✅ HIGH CONFIDENCE - Simulation is very reliable\")\n",
    "        elif validation_result.overall_score > 0.6:\n",
    "            print(\"⚠️  MEDIUM CONFIDENCE - Simulation has some reliability\")\n",
    "        else:\n",
    "            print(\"❌ LOW CONFIDENCE - Simulation may not be reliable\")\n",
    "    else:\n",
    "        print(\"⚠️  Validation completed but confidence score could not be calculated\")\n",
    "        print(f\"📈 Summary: {validation_result.summary}\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(\"❌ VALIDATION ERROR\")\n",
    "    print(\"=\" * 50)\n",
    "    print(f\"Error during validation: {str(e)}\")\n",
    "    print(\"💡 This might be due to insufficient data or statistical computation issues.\")\n",
    "    print(\"   Consider using more data points or different validation methods.\")\n",
    "    \n",
    "    # Create a fallback basic comparison\n",
    "    print(\"\\n📊 FALLBACK: Basic Data Comparison\")\n",
    "    print(\"=\" * 40)\n",
    "    \n",
    "    # Simple mean comparisons\n",
    "    real_conv = np.mean(real_customer_data['key_results']['conversion_rate'])\n",
    "    sim_conv = np.mean(tinytroupe_simulation_data['key_results']['conversion_rate'])\n",
    "    \n",
    "    print(f\"Real conversion rate: {real_conv:.1%}\")\n",
    "    print(f\"Simulated conversion rate: {sim_conv:.1%}\")\n",
    "    print(f\"Predicted improvement: {((sim_conv - real_conv) / real_conv) * 100:.1f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c287a55e",
   "metadata": {},
   "source": [
    "## Deep Dive: What the Numbers Tell Us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "35224dd8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-14T03:50:42.083710Z",
     "iopub.status.busy": "2025-07-14T03:50:42.082712Z",
     "iopub.status.idle": "2025-07-14T03:50:42.114524Z",
     "shell.execute_reply": "2025-07-14T03:50:42.110406Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📊 STATISTICAL ANALYSIS\n",
      "========================================\n",
      "📈 Metrics analyzed: customer_satisfaction, cart_abandonment_rate, conversion_rate, overall_revenue_per_visitor, average_order_value\n",
      "📊 Statistical analysis encountered an issue:\n",
      "   'str' object has no attribute 'get'\n",
      "   This might be due to insufficient data for statistical testing.\n"
     ]
    }
   ],
   "source": [
    "# Examine the statistical evidence\n",
    "try:\n",
    "    if validation_result.statistical_results:\n",
    "        print(\"📊 STATISTICAL ANALYSIS\")\n",
    "        print(\"=\" * 40)\n",
    "        \n",
    "        if \"error\" in validation_result.statistical_results:\n",
    "            print(f\"❌ Error: {validation_result.statistical_results['error']}\")\n",
    "        else:\n",
    "            metrics = validation_result.statistical_results['common_metrics']\n",
    "            print(f\"📈 Metrics analyzed: {', '.join(metrics)}\")\n",
    "            \n",
    "            # Show key findings\n",
    "            test_results = validation_result.statistical_results['test_results']\n",
    "            significant_differences = []\n",
    "            \n",
    "            for treatment_name, treatment_results in test_results.items():\n",
    "                for metric, metric_results in treatment_results.items():\n",
    "                    for test_name, test_result in metric_results.items():\n",
    "                        if test_result.get('significant', False):\n",
    "                            p_val = test_result.get('p_value', 'N/A')\n",
    "                            if isinstance(p_val, (int, float)) and not np.isnan(p_val):\n",
    "                                significant_differences.append(f\"{metric}: p={p_val:.3f}\")\n",
    "                            else:\n",
    "                                significant_differences.append(f\"{metric}: significant\")\n",
    "            \n",
    "            if significant_differences:\n",
    "                print(\"⚠️  Significant differences found:\")\n",
    "                for diff in significant_differences:\n",
    "                    print(f\"   • {diff}\")\n",
    "            else:\n",
    "                print(\"✅ No significant differences - simulation aligns with real data\")\n",
    "    else:\n",
    "        print(\"📊 No statistical results available\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(\"📊 Statistical analysis encountered an issue:\")\n",
    "    print(f\"   {str(e)}\")\n",
    "    print(\"   This might be due to insufficient data for statistical testing.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e633ae81",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-14T03:50:42.128165Z",
     "iopub.status.busy": "2025-07-14T03:50:42.127152Z",
     "iopub.status.idle": "2025-07-14T03:50:42.157944Z",
     "shell.execute_reply": "2025-07-14T03:50:42.155909Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🧠 REASONING ANALYSIS\n",
      "========================================\n",
      "🎯 Reasoning alignment: 28.2%\n",
      "❌ Agent reasoning differs significantly from real customers\n",
      "\n",
      "📝 Summary comparison: 40.0% similar\n",
      "💡 Key insight: The two texts discuss the concept of checkout processes but from opposing perspectives. The first text highlights the is...\n"
     ]
    }
   ],
   "source": [
    "# Examine the reasoning alignment\n",
    "try:\n",
    "    if validation_result.semantic_results:\n",
    "        print(\"\\n🧠 REASONING ANALYSIS\")\n",
    "        print(\"=\" * 40)\n",
    "        \n",
    "        avg_proximity = validation_result.semantic_results.get('average_proximity')\n",
    "        if avg_proximity and not np.isnan(avg_proximity):\n",
    "            print(f\"🎯 Reasoning alignment: {avg_proximity:.1%}\")\n",
    "            \n",
    "            if avg_proximity > 0.7:\n",
    "                print(\"✅ Agent reasoning closely matches real customer thinking\")\n",
    "            elif avg_proximity > 0.5:\n",
    "                print(\"⚠️  Agent reasoning somewhat matches real customer thinking\")\n",
    "            else:\n",
    "                print(\"❌ Agent reasoning differs significantly from real customers\")\n",
    "        else:\n",
    "            print(\"⚠️  Could not calculate reasoning alignment score\")\n",
    "        \n",
    "        # Show reasoning comparison\n",
    "        summary_comp = validation_result.semantic_results.get('summary_comparison')\n",
    "        if summary_comp and summary_comp.get('proximity_score') and not np.isnan(summary_comp.get('proximity_score', 0)):\n",
    "            print(f\"\\n📝 Summary comparison: {summary_comp['proximity_score']:.1%} similar\")\n",
    "            justification = summary_comp.get('justification', '')\n",
    "            if justification:\n",
    "                print(f\"💡 Key insight: {justification[:120]}...\")\n",
    "        else:\n",
    "            print(\"\\n📝 Summary comparison: Unable to calculate similarity\")\n",
    "    else:\n",
    "        print(\"\\n🧠 No semantic analysis results available\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(\"\\n🧠 Semantic analysis encountered an issue:\")\n",
    "    print(f\"   {str(e)}\")\n",
    "    print(\"   This might be due to missing justification data or semantic processing issues.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eaf5e1d",
   "metadata": {},
   "source": [
    "## Business Impact Assessment\n",
    "\n",
    "Based on the validation results, here's what this means for your $2M investment decision:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "085eb64a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-14T03:50:42.169977Z",
     "iopub.status.busy": "2025-07-14T03:50:42.168983Z",
     "iopub.status.idle": "2025-07-14T03:51:19.615149Z",
     "shell.execute_reply": "2025-07-14T03:51:19.614243Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💼 BUSINESS IMPACT ASSESSMENT\n",
      "==================================================\n",
      "🎯 Simulation Confidence: 33.2%\n",
      "⚠️  Risk Level: HIGH\n",
      "📊 Risk-Adjusted Revenue Increase: 20.4%\n",
      "💰 Expected Monthly Revenue Gain: $204,000\n",
      "📈 Expected Annual Revenue Gain: $2,448,000\n",
      "🎯 Investment Payback Period: 9.8 months\n",
      "\n",
      "🏆 RECOMMENDATION: CONSIDER more validation\n"
     ]
    }
   ],
   "source": [
    "# Calculate business impact based on validation confidence\n",
    "try:\n",
    "    confidence_score = validation_result.overall_score if validation_result.overall_score is not None else 0.5\n",
    "    predicted_revenue_increase = 0.68  # 68% increase from simulation\n",
    "    current_monthly_revenue = 1000000  # $1M per month\n",
    "\n",
    "    print(\"💼 BUSINESS IMPACT ASSESSMENT\")\n",
    "    print(\"=\" * 50)\n",
    "\n",
    "    # Risk-adjusted projections\n",
    "    if confidence_score > 0.8:\n",
    "        risk_adjustment = 0.9  # High confidence = 90% of predicted benefit\n",
    "        recommendation = \"PROCEED with investment\"\n",
    "        risk_level = \"LOW\"\n",
    "    elif confidence_score > 0.6:\n",
    "        risk_adjustment = 0.6  # Medium confidence = 60% of predicted benefit\n",
    "        recommendation = \"PROCEED with caution\"\n",
    "        risk_level = \"MEDIUM\"\n",
    "    else:\n",
    "        risk_adjustment = 0.3  # Low confidence = 30% of predicted benefit\n",
    "        recommendation = \"CONSIDER more validation\"\n",
    "        risk_level = \"HIGH\"\n",
    "\n",
    "    expected_revenue_increase = predicted_revenue_increase * risk_adjustment\n",
    "    monthly_revenue_gain = current_monthly_revenue * expected_revenue_increase\n",
    "    annual_revenue_gain = monthly_revenue_gain * 12\n",
    "\n",
    "    print(f\"🎯 Simulation Confidence: {confidence_score:.1%}\")\n",
    "    print(f\"⚠️  Risk Level: {risk_level}\")\n",
    "    print(f\"📊 Risk-Adjusted Revenue Increase: {expected_revenue_increase:.1%}\")\n",
    "    print(f\"💰 Expected Monthly Revenue Gain: ${monthly_revenue_gain:,.0f}\")\n",
    "    print(f\"📈 Expected Annual Revenue Gain: ${annual_revenue_gain:,.0f}\")\n",
    "    \n",
    "    if monthly_revenue_gain > 0:\n",
    "        print(f\"🎯 Investment Payback Period: {2000000 / monthly_revenue_gain:.1f} months\")\n",
    "    else:\n",
    "        print(\"🎯 Investment Payback Period: Cannot calculate (no expected gain)\")\n",
    "        \n",
    "    print(f\"\\n🏆 RECOMMENDATION: {recommendation}\")\n",
    "    \n",
    "except Exception as e:\n",
    "    print(\"💼 BUSINESS IMPACT ASSESSMENT\")\n",
    "    print(\"=\" * 50)\n",
    "    print(f\"❌ Error calculating business impact: {str(e)}\")\n",
    "    print(\"💡 Using basic assessment based on available data...\")\n",
    "    \n",
    "    # Basic fallback assessment\n",
    "    print(f\"🎯 Predicted Revenue Increase: 68%\")\n",
    "    print(f\"💰 Investment Amount: $2M\")\n",
    "    print(f\"⚠️  Recommendation: Proceed with caution due to validation issues\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "719d36aa",
   "metadata": {},
   "source": [
    "## Quick Example: Statistics-Only Validation\n",
    "\n",
    "For simpler cases where you only have metrics (no customer reasoning), you can run statistical validation only:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "789e5b32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⚡ Quick validation score: 16.6%\n",
      "📊 Results: Statistical validation: 2/2 tests significant, average effect size: 5.170; Overall validation score: 0.166\n"
     ]
    }
   ],
   "source": [
    "# Simple metrics-only validation\n",
    "simple_real_data = {\n",
    "    \"name\": \"Real Data - Metrics Only\",\n",
    "    \"key_results\": {\n",
    "        # Extended data for more robust statistical testing\n",
    "        \"click_through_rate\": [0.12, 0.15, 0.13, 0.14, 0.11, 0.16, 0.13, 0.12, 0.15, 0.14, \n",
    "                              0.13, 0.15, 0.12, 0.14, 0.16, 0.13, 0.12, 0.15, 0.14, 0.13],\n",
    "        \"time_on_page\": [45, 52, 48, 50, 46, 54, 47, 49, 51, 48, \n",
    "                        46, 53, 49, 47, 52, 50, 48, 51, 49, 47]  # seconds\n",
    "    },\n",
    "    \"result_types\": {\n",
    "        \"click_through_rate\": \"per_agent\",\n",
    "        \"time_on_page\": \"per_agent\"\n",
    "    }\n",
    "}\n",
    "\n",
    "simple_simulation_data = {\n",
    "    \"name\": \"Simulation - Metrics Only\", \n",
    "    \"key_results\": {\n",
    "        \"click_through_rate\": [0.18, 0.21, 0.19, 0.20, 0.17, 0.22, 0.19, 0.18, 0.21, 0.20,\n",
    "                              0.19, 0.21, 0.18, 0.20, 0.22, 0.19, 0.18, 0.21, 0.20, 0.19],\n",
    "        \"time_on_page\": [62, 68, 65, 64, 61, 70, 63, 66, 67, 64,\n",
    "                        61, 69, 65, 63, 68, 66, 62, 67, 64, 63]  # seconds\n",
    "    },\n",
    "    \"result_types\": {\n",
    "        \"click_through_rate\": \"per_agent\",\n",
    "        \"time_on_page\": \"per_agent\"\n",
    "    }\n",
    "}\n",
    "\n",
    "# Quick statistical validation with error handling\n",
    "try:\n",
    "    quick_result = validate_simulation_experiment_empirically(\n",
    "        control_data=simple_real_data,\n",
    "        treatment_data=simple_simulation_data,\n",
    "        validation_types=[\"statistical\"],\n",
    "        output_format=\"values\"\n",
    "    )\n",
    "    \n",
    "    if quick_result.overall_score is not None:\n",
    "        print(f\"⚡ Quick validation score: {quick_result.overall_score:.1%}\")\n",
    "        print(f\"📊 Results: {quick_result.summary}\")\n",
    "    else:\n",
    "        print(\"⚡ Quick validation completed with issues\")\n",
    "        print(f\"📊 Results: {quick_result.summary}\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(f\"⚡ Quick validation error: {str(e)}\")\n",
    "    print(\"📊 Fallback: Basic comparison shows simulation predicts ~50% improvement\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd9b10c7",
   "metadata": {},
   "source": [
    "## Key Takeaways\n",
    "\n",
    "This validation system gives you confidence in your TinyTroupe simulations by:\n",
    "\n",
    "1. **📊 Statistical Validation** - Tests if your simulation results are statistically similar to real-world data\n",
    "2. **🧠 Semantic Validation** - Compares agent reasoning with real customer thinking patterns  \n",
    "3. **🎯 Confidence Scoring** - Provides a clear 0-100% confidence score for business decisions\n",
    "4. **💰 Risk Assessment** - Helps you make informed investment decisions based on validation confidence\n",
    "\n",
    "**The Bottom Line**: Before making expensive business decisions based on simulations, validate them against real data. This system tells you exactly how much you can trust your TinyTroupe predictions.\n",
    "\n",
    "### Next Steps\n",
    "- Collect real customer data for your use case\n",
    "- Run your TinyTroupe simulation  \n",
    "- Use this validation system to assess confidence\n",
    "- Make data-driven business decisions\n",
    "\n",
    "**Remember**: A simulation is only as good as its validation against reality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e9aaad6",
   "metadata": {},
   "source": [
    "# Advanced Feature: Categorical Data Validation\n",
    "\n",
    "**New Capability**: The validation system now supports categorical string data directly - no manual encoding required!\n",
    "\n",
    "**Key Benefits**:\n",
    "- 🎯 **Direct String Input** - Use categories like [\"yes\", \"no\", \"maybe\"] or [\"low\", \"medium\", \"high\"] directly\n",
    "- 🔄 **Automatic Conversion** - Strings are automatically normalized and converted to ordinal values\n",
    "- 📊 **KS Test Support** - Compare distributions, not just means, using the Kolmogorov-Smirnov test\n",
    "- 📋 **Categorical Reports** - Get comprehensive reports showing category mappings and distributions\n",
    "\n",
    "This is especially useful for survey responses, preference studies, and qualitative research validation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a890a87",
   "metadata": {},
   "source": [
    "## Example: Product Preference Study\n",
    "\n",
    "**Scenario**: Your company wants to validate how TinyTroupe agents respond to product preferences compared to real humans.\n",
    "\n",
    "**Real Data**: Human survey responses (mostly uncertain/neutral)  \n",
    "**Simulation**: AI agent responses (more decisive/polarized)  \n",
    "**Question**: Do the response patterns match well enough to trust the simulation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06444952",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define categorical data - notice we use strings directly, no manual encoding!\n",
    "\n",
    "# Control data: Human responses (mostly neutral/uncertain)\n",
    "human_responses = {\n",
    "    \"name\": \"Human Control Group\",\n",
    "    \"description\": \"Empirical human responses to product preference\",\n",
    "    \"key_results\": {\n",
    "        \"preference\": [\n",
    "            \"no\", \"maybe\", \"maybe\", \"yes\", \"maybe\", \n",
    "            \"no\", \"maybe\", \"maybe\", \"yes\", \"maybe\",\n",
    "            \"no\", \"maybe\", \"yes\", \"maybe\", \"maybe\"\n",
    "        ],\n",
    "        \"satisfaction\": [\n",
    "            \"low\", \"medium\", \"medium\", \"high\", \"medium\",\n",
    "            \"low\", \"medium\", \"medium\", \"high\", \"medium\", \n",
    "            \"medium\", \"medium\", \"high\", \"medium\", \"medium\"\n",
    "        ]\n",
    "    },\n",
    "    \"result_types\": {\n",
    "        \"preference\": \"per_agent\",\n",
    "        \"satisfaction\": \"per_agent\"\n",
    "    },\n",
    "    \"agent_justifications\": [\n",
    "        \"Price seems reasonable but I'm not sure about quality\",\n",
    "        \"Need more information before deciding\",\n",
    "        \"Looks good but want to compare alternatives\"\n",
    "    ],\n",
    "    \"justification_summary\": \"Humans showed uncertainty and wanted more information before deciding\"\n",
    "}\n",
    "\n",
    "# Treatment data: AI agent responses (more decisive/polarized)\n",
    "ai_responses = {\n",
    "    \"name\": \"AI Agent Group\", \n",
    "    \"description\": \"AI agent simulation results for product preference\",\n",
    "    \"key_results\": {\n",
    "        \"preference\": [\n",
    "            \"yes\", \"yes\", \"no\", \"yes\", \"yes\",\n",
    "            \"no\", \"yes\", \"yes\", \"no\", \"yes\",\n",
    "            \"yes\", \"no\", \"yes\", \"yes\", \"yes\"\n",
    "        ],\n",
    "        \"satisfaction\": [\n",
    "            \"high\", \"high\", \"low\", \"high\", \"high\",\n",
    "            \"low\", \"high\", \"high\", \"low\", \"high\",\n",
    "            \"high\", \"low\", \"high\", \"high\", \"high\"  \n",
    "        ]\n",
    "    },\n",
    "    \"result_types\": {\n",
    "        \"preference\": \"per_agent\", \n",
    "        \"satisfaction\": \"per_agent\"\n",
    "    },\n",
    "    \"agent_justifications\": [\n",
    "        \"Clear value proposition with good price-to-quality ratio\",\n",
    "        \"Product specs don't meet my requirements\",\n",
    "        \"Excellent features that justify the cost\"\n",
    "    ],\n",
    "    \"justification_summary\": \"AI agents made more decisive judgments based on clear criteria\"\n",
    "}\n",
    "\n",
    "print(\"📊 Data loaded with categorical string values:\")\n",
    "print(\"Human preferences:\", human_responses[\"key_results\"][\"preference\"])\n",
    "print(\"AI preferences:\", ai_responses[\"key_results\"][\"preference\"])\n",
    "print(\"\\n✨ Notice: No manual encoding required - strings are used directly!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac522956",
   "metadata": {},
   "source": [
    "## Comparison: T-test vs KS Test for Categorical Data\n",
    "\n",
    "The traditional t-test compares **means**, but for categorical data, we often care about **distributions**. \n",
    "\n",
    "The **Kolmogorov-Smirnov (KS) test** compares entire distributions and can detect differences that t-tests miss.\n",
    "\n",
    "Let's see both in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8bda571",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Analysis 1: Traditional Welch t-test (comparing means)\n",
    "print(\"📈 Analysis 1: Traditional Welch t-test (comparing means)\")\n",
    "print(\"-\" * 50)\n",
    "\n",
    "try:\n",
    "    result_ttest = validate_simulation_experiment_empirically(\n",
    "        control_data=human_responses,\n",
    "        treatment_data=ai_responses,\n",
    "        validation_types=[\"statistical\"],\n",
    "        statistical_test_type=\"welch_t_test\",  # Default test type\n",
    "        output_format=\"values\"\n",
    "    )\n",
    "    \n",
    "    if result_ttest.statistical_results and \"test_results\" in result_ttest.statistical_results:\n",
    "        for metric in [\"preference\", \"satisfaction\"]:\n",
    "            test_result = result_ttest.statistical_results[\"test_results\"][\"treatment\"][metric]\n",
    "            print(f\"{metric.title()}:\")\n",
    "            print(f\"  - Significant: {test_result.get('significant', 'N/A')}\")\n",
    "            print(f\"  - p-value: {test_result.get('p_value', 'N/A'):.4f}\" if isinstance(test_result.get('p_value'), (int, float)) else f\"  - p-value: {test_result.get('p_value', 'N/A')}\")\n",
    "            print(f\"  - Effect size: {test_result.get('effect_size', 'N/A')}\")\n",
    "    else:\n",
    "        print(\"T-test results not available - may be due to data limitations\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(f\"T-test analysis error: {str(e)}\")\n",
    "    print(\"This might happen with categorical data - KS test is often more appropriate\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91836d3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Analysis 2: Kolmogorov-Smirnov test (comparing distributions)\n",
    "print(\"\\n📊 Analysis 2: Kolmogorov-Smirnov test (comparing distributions)\")\n",
    "print(\"-\" * 50)\n",
    "\n",
    "try:\n",
    "    result_ks = validate_simulation_experiment_empirically(\n",
    "        control_data=human_responses,\n",
    "        treatment_data=ai_responses,\n",
    "        validation_types=[\"statistical\"],\n",
    "        statistical_test_type=\"ks_test\",  # Use KS test instead\n",
    "        output_format=\"values\"\n",
    "    )\n",
    "    \n",
    "    if result_ks.statistical_results and \"test_results\" in result_ks.statistical_results:\n",
    "        for metric in [\"preference\", \"satisfaction\"]:\n",
    "            test_result = result_ks.statistical_results[\"test_results\"][\"treatment\"][metric]\n",
    "            print(f\"{metric.title()}:\")\n",
    "            print(f\"  - Significant: {test_result.get('significant', 'N/A')}\")\n",
    "            print(f\"  - p-value: {test_result.get('p_value', 'N/A'):.4f}\" if isinstance(test_result.get('p_value'), (int, float)) else f\"  - p-value: {test_result.get('p_value', 'N/A')}\")\n",
    "            print(f\"  - KS statistic: {test_result.get('ks_statistic', 'N/A'):.4f}\" if isinstance(test_result.get('ks_statistic'), (int, float)) else f\"  - KS statistic: {test_result.get('ks_statistic', 'N/A')}\")\n",
    "            print(f\"  - Interpretation: {test_result.get('interpretation', 'N/A')}\")\n",
    "    else:\n",
    "        print(\"KS test results not available\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(f\"KS test analysis error: {str(e)}\")\n",
    "    \n",
    "print(\"\\n💡 Key Insight: KS test is often better for categorical data because it\")\n",
    "print(\"   compares the entire distribution shape, not just the average values!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab95819e",
   "metadata": {},
   "source": [
    "## Behind the Scenes: Categorical Data Conversion\n",
    "\n",
    "The system automatically converts string categories to ordinal values for statistical analysis, while preserving the original categories for interpretation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "028aeffb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show how categorical data is automatically converted\n",
    "from tinytroupe.validation.simulation_validator import SimulationExperimentDataset\n",
    "\n",
    "print(\"🔄 Categorical Data Conversion Details\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "# Create dataset to inspect mappings\n",
    "human_dataset = SimulationExperimentDataset.parse_obj(human_responses)\n",
    "\n",
    "for metric in [\"preference\", \"satisfaction\"]:\n",
    "    print(f\"\\n{metric.title()} Categories:\")\n",
    "    \n",
    "    # Show the automatic mapping\n",
    "    categories = human_dataset.get_categorical_values(metric)\n",
    "    mapping = human_dataset.categorical_mappings[metric]\n",
    "    \n",
    "    for category in sorted(categories):\n",
    "        ordinal = mapping[category]\n",
    "        print(f\"  '{category}' → {ordinal}\")\n",
    "    \n",
    "    # Show distribution in both groups\n",
    "    summary = human_dataset.get_metric_summary(metric)\n",
    "    if \"category_distribution\" in summary:\n",
    "        print(f\"\\n{metric.title()} Distribution - Humans:\")\n",
    "        for category, count in summary[\"category_distribution\"].items():\n",
    "            percentage = count / len(human_responses[\"key_results\"][metric]) * 100\n",
    "            print(f\"  {category}: {count} ({percentage:.1f}%)\")\n",
    "    \n",
    "    # Compare with AI responses\n",
    "    ai_dataset = SimulationExperimentDataset.parse_obj(ai_responses)\n",
    "    ai_summary = ai_dataset.get_metric_summary(metric)\n",
    "    if \"category_distribution\" in ai_summary:\n",
    "        print(f\"\\n{metric.title()} Distribution - AI:\")\n",
    "        for category, count in ai_summary[\"category_distribution\"].items():\n",
    "            percentage = count / len(ai_responses[\"key_results\"][metric]) * 100\n",
    "            print(f\"  {category}: {count} ({percentage:.1f}%)\")\n",
    "\n",
    "print(\"\\n✨ Key Benefits:\")\n",
    "print(\"  • Automatic normalization (lowercasing, whitespace removal)\")\n",
    "print(\"  • Consistent ordinal mapping across groups\")\n",
    "print(\"  • Original categories preserved for reports\")\n",
    "print(\"  • No manual encoding required!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85153939",
   "metadata": {},
   "source": [
    "## Comprehensive Report with Categorical Information\n",
    "\n",
    "Generate a full validation report that includes categorical data details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "798c09c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate comprehensive report with categorical information\n",
    "print(\"📋 Generating Comprehensive Report with Categorical Information\")\n",
    "print(\"-\" * 50)\n",
    "\n",
    "try:\n",
    "    comprehensive_report = validate_simulation_experiment_empirically(\n",
    "        control_data=human_responses,\n",
    "        treatment_data=ai_responses,\n",
    "        validation_types=[\"statistical\", \"semantic\"],\n",
    "        statistical_test_type=\"ks_test\",\n",
    "        output_format=\"report\"  # Get full markdown report\n",
    "    )\n",
    "    \n",
    "    # Save report to file for inspection\n",
    "    report_filename = \"categorical_validation_report.md\"\n",
    "    with open(report_filename, \"w\", encoding=\"utf-8\") as f:\n",
    "        f.write(comprehensive_report)\n",
    "    \n",
    "    print(f\"✅ Full report saved to '{report_filename}'\")\n",
    "    \n",
    "    # Show a preview of the report\n",
    "    print(\"\\n📄 Report Preview (first 500 characters):\")\n",
    "    print(\"-\" * 30)\n",
    "    print(comprehensive_report[:500] + \"...\" if len(comprehensive_report) > 500 else comprehensive_report)\n",
    "    \n",
    "except Exception as e:\n",
    "    print(f\"❌ Error generating report: {str(e)}\")\n",
    "    print(\"This might be due to semantic analysis limitations with the current setup\")\n",
    "    \n",
    "    # Try statistical-only report as fallback\n",
    "    try:\n",
    "        print(\"\\n🔄 Generating statistical-only report as fallback...\")\n",
    "        stats_report = validate_simulation_experiment_empirically(\n",
    "            control_data=human_responses,\n",
    "            treatment_data=ai_responses,\n",
    "            validation_types=[\"statistical\"],\n",
    "            statistical_test_type=\"ks_test\",\n",
    "            output_format=\"report\"\n",
    "        )\n",
    "        \n",
    "        with open(\"categorical_stats_report.md\", \"w\", encoding=\"utf-8\") as f:\n",
    "            f.write(stats_report)\n",
    "        \n",
    "        print(\"✅ Statistical report saved to 'categorical_stats_report.md'\")\n",
    "        print(\"\\n📄 Statistical Report Preview:\")\n",
    "        print(\"-\" * 30)\n",
    "        print(stats_report[:400] + \"...\" if len(stats_report) > 400 else stats_report)\n",
    "        \n",
    "    except Exception as fallback_e:\n",
    "        print(f\"❌ Fallback also failed: {str(fallback_e)}\")\n",
    "        print(\"Manual report generation may be needed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874a8e4e",
   "metadata": {},
   "source": [
    "## Why KS Test is Better for Categorical Data\n",
    "\n",
    "Here's a demonstration of why the KS test can be superior to t-tests for categorical data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0af8f0f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scenario: Same \"average\" response but different distributions\n",
    "print(\"🎯 Demonstrating KS Test Advantages for Categorical Data\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "print(\"\\nScenario: Same 'average' response but different distributions\")\n",
    "\n",
    "# Both groups have same mean (1.0) but different distributions\n",
    "control_uniform = {\n",
    "    \"name\": \"Uniform Distribution\",\n",
    "    \"key_results\": {\n",
    "        \"response\": [\"low\", \"medium\", \"high\"] * 5  # Even distribution: 5 low, 5 medium, 5 high\n",
    "    },\n",
    "    \"result_types\": {\"response\": \"per_agent\"}\n",
    "}\n",
    "\n",
    "treatment_polarized = {\n",
    "    \"name\": \"Polarized Distribution\", \n",
    "    \"key_results\": {\n",
    "        \"response\": [\"low\"] * 7 + [\"high\"] * 8  # Polarized: 7 low, 0 medium, 8 high (same mean ≈ 1.0)\n",
    "    },\n",
    "    \"result_types\": {\"response\": \"per_agent\"}\n",
    "}\n",
    "\n",
    "print(f\"\\nControl (Uniform): {control_uniform['key_results']['response']}\")\n",
    "print(f\"Treatment (Polarized): {treatment_polarized['key_results']['response']}\")\n",
    "\n",
    "# Calculate means to show they're similar\n",
    "from tinytroupe.validation.simulation_validator import SimulationExperimentDataset\n",
    "control_ds = SimulationExperimentDataset.parse_obj(control_uniform)\n",
    "treatment_ds = SimulationExperimentDataset.parse_obj(treatment_polarized)\n",
    "\n",
    "control_mean = sum(control_ds.get_metric_values(\"response\")) / len(control_ds.get_metric_values(\"response\"))\n",
    "treatment_mean = sum(treatment_ds.get_metric_values(\"response\")) / len(treatment_ds.get_metric_values(\"response\"))\n",
    "\n",
    "print(f\"\\nMeans (after conversion):\")\n",
    "print(f\"Control mean: {control_mean:.2f}\")\n",
    "print(f\"Treatment mean: {treatment_mean:.2f}\")\n",
    "print(f\"Mean difference: {abs(control_mean - treatment_mean):.2f} (very small!)\")\n",
    "\n",
    "# Compare t-test vs KS test\n",
    "print(f\"\\n📊 Statistical Test Comparison:\")\n",
    "print(\"-\" * 40)\n",
    "\n",
    "try:\n",
    "    # T-test (comparing means)\n",
    "    result_ttest = validate_simulation_experiment_empirically(\n",
    "        control_data=control_uniform,\n",
    "        treatment_data=treatment_polarized,\n",
    "        validation_types=[\"statistical\"],\n",
    "        statistical_test_type=\"welch_t_test\",\n",
    "        output_format=\"values\"\n",
    "    )\n",
    "    \n",
    "    # KS test (comparing distributions)\n",
    "    result_ks = validate_simulation_experiment_empirically(\n",
    "        control_data=control_uniform,\n",
    "        treatment_data=treatment_polarized,\n",
    "        validation_types=[\"statistical\"],\n",
    "        statistical_test_type=\"ks_test\",\n",
    "        output_format=\"values\"\n",
    "    )\n",
    "    \n",
    "    if (result_ttest.statistical_results and result_ks.statistical_results and \n",
    "        \"test_results\" in result_ttest.statistical_results and \n",
    "        \"test_results\" in result_ks.statistical_results):\n",
    "        \n",
    "        ttest_result = result_ttest.statistical_results[\"test_results\"][\"treatment\"][\"response\"]\n",
    "        ks_result = result_ks.statistical_results[\"test_results\"][\"treatment\"][\"response\"]\n",
    "        \n",
    "        print(f\"T-test (means):        Significant: {ttest_result.get('significant', 'N/A')}, p-value: {ttest_result.get('p_value', 'N/A'):.4f}\" if isinstance(ttest_result.get('p_value'), (int, float)) else f\"T-test (means):        Significant: {ttest_result.get('significant', 'N/A')}, p-value: {ttest_result.get('p_value', 'N/A')}\")\n",
    "        print(f\"KS test (distributions): Significant: {ks_result.get('significant', 'N/A')}, p-value: {ks_result.get('p_value', 'N/A'):.4f}\" if isinstance(ks_result.get('p_value'), (int, float)) else f\"KS test (distributions): Significant: {ks_result.get('significant', 'N/A')}, p-value: {ks_result.get('p_value', 'N/A')}\")\n",
    "        \n",
    "        print(f\"\\n💡 Insight: KS test can detect distributional differences that t-tests miss!\")\n",
    "        print(f\"   Even with similar means, the response patterns are fundamentally different.\")\n",
    "    else:\n",
    "        print(\"Test results not available - using fallback comparison\")\n",
    "        print(\"💡 Key Point: T-test focuses on means, KS test compares entire distributions\")\n",
    "        \n",
    "except Exception as e:\n",
    "    print(f\"Error in comparison: {str(e)}\")\n",
    "    print(\"💡 Key Point: T-test focuses on means, KS test compares entire distributions\")\n",
    "    print(\"   For categorical data, distributions often matter more than averages!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e2ac70",
   "metadata": {},
   "source": [
    "## Categorical Data Validation - Key Takeaways\n",
    "\n",
    "🎉 **Categorical data validation is now easy and powerful!**\n",
    "\n",
    "### ✅ Key Benefits:\n",
    "\n",
    "1. **🎯 Direct String Input** - No manual encoding required\n",
    "   - Use `[\"yes\", \"no\", \"maybe\"]` or `[\"low\", \"medium\", \"high\"]` directly\n",
    "   - Automatic normalization handles case and whitespace differences\n",
    "\n",
    "2. **🔄 Smart Conversion** - Automatic ordinal mapping\n",
    "   - Strings converted to meaningful ordinal values for analysis\n",
    "   - Original categories preserved for reports and interpretation\n",
    "\n",
    "3. **📊 Better Statistical Tests** - KS test for distributions\n",
    "   - Compare entire response patterns, not just averages\n",
    "   - Detect differences in behavior that t-tests might miss\n",
    "\n",
    "4. **📋 Rich Reports** - Categorical information included\n",
    "   - Category mappings and distributions in validation reports\n",
    "   - Easy to understand and share with stakeholders\n",
    "\n",
    "### 🚀 Best Practices:\n",
    "\n",
    "- **Use KS test** (`statistical_test_type=\"ks_test\"`) for categorical data\n",
    "- **Include semantic validation** when you have justification data\n",
    "- **Generate reports** (`output_format=\"report\"`) for comprehensive documentation\n",
    "- **Review category distributions** to understand behavior patterns\n",
    "\n",
    "### 💡 When to Use:\n",
    "\n",
    "- Survey responses and questionnaires\n",
    "- Product preferences and ratings\n",
    "- Qualitative research validation\n",
    "- Any time you have string-based categorical data\n",
    "\n",
    "This makes TinyTroupe validation accessible for qualitative research and survey-based studies!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py310",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
